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ABSTRACT t , 

Traditionally, outcomes measured by postsecondary 

vocational education assessment have included labor force 

participation rates, type of employment, earnings, and employer 

satisfaction with employees. Today, however, legislators and 

policymakers are moving more and more toward accountability measures 

tied to students' educational outcomes, such as basic educational 

skills, higher-order thinking skills, knowledge of the world of work, 

and continuing-education rates. New approaches for assessing programs 

m ay include the following: (1) Total Quality Management (TQM) which 

focuses on evaluation for the sake of improvement; (2; the student 

success m-del, which strives to create an institutional environment 

that contributes to developing and maintaining student success and 

encourages students to set high standards, document their goals and 

intended outcomes, and monitor their outcomes; (3) value-added 

assessment, which involves gathering baseline information about 

entry-level competency and compare it with exit-level performance, 

(4) concept mapping, an approach that enables diverse groups of 

practitioners to describe how their programs work and then to use 

that collective wisdom to articulate how the program is intended to 

influence outcomes; and (5) performance assessment, an umbrella term 

that describes a variety of techniques designed to assess directly 

and "authentically" what students know and can do in a form other 

than paper and pencil or multiple choice tests. These choices of 

assessment are not necessarily exclusive but can be used in 

connection with each other. (KC) 
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Alternative Approaches to 
Outcomes Assessment for 
Postsecondary Vocational 
Education 

At least since the early 1980s, policy- 
makers, business leaders, and educa- 
tional reformers have called for in- 
creased performance and accountabil- 
ity in education. The emphasis on ac- 
countability is intensifying. For 
example, bills awaiting action in Con- 
gress would provide individual states 
with large block grants for vocational 
education and job training with a great 
deal of flexibility and less federal pre- 
scriptions. But in at least one area — ac- 
countability — the plans call for even 
stricter federal control than in the 
past. 

As part of the general call for re- 
form, state governments and accre- 
diting associations have strongly rec- 
ommended the adoption of outcomes 
assessment in postsecondary educa- 
tion. 

Reinforcing the need for improve- 
ments, the Perkins Act of 1900 requires 
each state to measure student learning 
gains in basic and more advanced aca- 
demic skills as well as vocational skills. 
The requirement to measure academic 
competence represents a departure 
from the type of outcomes tradition- 
ally measured. 

"Traditional" outcomes are those 
that have been routinely used in post- 
secondary vocational education, such 
as job placement, occupational com- 
petence, program completion or re- 
tention, and earnings. Traditional out- 
comes are usually measured with certi- 
fication tests, follow-up surveys, and 
self reports. The goal of alternative as- 
sessment is to produce more valid in- 
formation about the results of voca- 
tional education programs than has 
- been available with more traditional 
testing procedures. 

This Center locus explores alternative 
wavs of assessing outcomes. But first it 
is important to clarify which outcomes 
to assess. Vocational programs have 
often been thought of solely in terms 
of students* economic outcomes, such 
as: 



• labor force participation rates 

• type of employment 

• earnings 

• employer satisfaction with employees 

However, legislators and policy- 
makers are moving more and more in- 
ward accountability measures tied to 
students' educational outcomes, such 

as: 

• basic educational skills 

• higher-order thinking skills 

• knowledge of the world of work 

• continuing-cducation rates 

One might suppose that the only 
important evaluation information is 
that related to student outcomes. How- 
ever, teachers and administrators need 
to know what worked, what did not 
work, and why. And practitioners, pro- 
gram managers, policymakers, and 
fundcrs need a basis for deciding 
whether to modify, enhance, or drop 
programs. Therefore, in addition to 
information on student outcomes, a 
broad, meaningful approach to assess- 
ment requires valid and reliable infor- 
mation on: 

• the veeds for vocational education as 
expressed by students, employers, 
and society, and 

• the educational processes employed 
in the programs. 

Needs, of course, provide the basis 
for evaluation, and process informa- 
tion helps evaluators understand and 
interpret what has occurred and why it 
occurred, and decide what needs to be 
changed. 

One needs also to look at two other 
broad categories of outcomes: institu- 
tional and program. There are certain 
goals that can be effeclivelv addressed 
only at the institutional level because 
these outcomes represent an aggre- 
gation of all efforts occurring across 
the institution, such as institution-wide 
retention and completion rates. But 
when a specific program represents 
only one part of a larger institution, 
outcomes must be assessed at the pro- 
gram level because that is where thev 
have the most meaning. 
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As the demands and expectations 
for outcomes assessment increase, so 
do the challenges for practitioners in 
postsecondary institutions. Post- 
secondary institutions and their voca- 
tional programs serve, several Different 
groups of students simultaneously. 
These include traditional, full-time 
students looking to transfer to four- 
vear colleges; young adults seeking 
higher-paying employment options; 
and middle-aged adults seeking re- 
training for career change. Each of 
these groups seeks different outcomes 
and requires different outcomes assess- 
ment. Since students who come to 
these schools with different needs en- 
counter the curriculum in different 
wavs and in different stages of their 
education, designing outcomes assess- 
ment around highly structured cur- 
riculum paths is extremely difficult to 
accomplish. 

Further, according to Ewell (1992), 
community college students "don't 
stand for assessment." He advocates 
assessment systems that arc infused 
into the curriculum so that students 
are not intimidated by formal testing 
procedures. Alternative assessment ap- 
proaches that enable students to dem- 
onstrate skills and knowledge in natu- 
ral settings and at convenient times 
and places can help to accomplish this 
goal. 

Kach of the following approaches of- 
fers a unique perspective for thinking 
about, collecting information on, and 
analv/.ing outcomes for postsecondary 
education. 

Total Quality Management (TQM) 

American businesses are adopting 
TQM because of their need to improve 
qualitv to increase economic competi- 
tiveness. TQM is now being contem- 
plated in relation to educational re- 
form. The fundamental goal of TQM 
is to identify, meet, and exceed cus- 
tomer needs by managing and improv- 
ing work processes to make continuous 
improvements and bv ensuring that 
even-one is involved in carrying out 
the initiative. With TQM. measure- 
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mutt is focused on the cm ire svslcin 
rather tiian simply on the final prod- 
uct. 

In a TQM approach, deviations 
from specified performance standards 
are analyzed and fed back into the sys- 
tem to make improvements. Although 
assessment to achieve accountability is 
part of this approach, it is not the pri- 
mary goal. The priinaiy goal — the 
critical purpose of TQM — is 10 create 
and support an environment that 
yields continuous quality improve- 
ment. 

Thus, TQM focuses on how work 
processes contribute to or hinder de- 
realization of customer expectations. A 
fully functioning TQM approach 
yields outputs (products and sen-ices) 
that are responsive to ever-changing 
customer needs. In an educational in- 
stitution, for example, the goal of 
TQM is to ensure that all work proc- 
esses contribute to meeting student 
needs and expectations and are con- 
tinually refined and re focused on 
meeting those needs. Further, accord- 
ing to TQM philosophy, a school's in- 
stitutional effectiveness can be deter- 
mined by measuring the effectiveness 
and efficiency of these processes. 

While difficult to generalize, a com- 
parison of TQM to other educational 
outcomes assessment approaches re- 
veals that TQM is more comprehen- 
sive in its approach and more highly 
focused on evaluation for the sake of 
improvement than for the sake ol ac- 
countability. Imjnovancnt is the kev 
concept. 

In 1991, Axland identified fumtccn 
community and technical colleges in- 
voked with TQM, of which roughlv 
two-thirds were implementing TQM 
within their own administrative units. 
Fox Valley Technical College (F\TC) 
in Appleton, Wisconsin has been using 
TQM since 1980. Comment cards stra- 
tegically located on campus and 
picked up two or three limes a week 
provide the College a direct and 
anonvinous mechanism for monitor- 
ing and managing student satisfaction. 
The College also uses focus groups to 
collect data on student needs and ex- 
pectations. In addition, problem-solv- 
ing teams are lortued within units 
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across the campus to enable employees 
to take corrective action when prob- 
lems are detected. Teams directly con- 
front problems that could poteniialK 
jeopardize students" positive out- 
comes, FVf'C is so committed to using 
measurement to achieve improve- 
ments that it offers educational guar- 
antees to its graduates and offers re- 
training or refunds if these guarantees 
are not met. 

The Student Success Model 

The goal of the student success 
model is to ( reate ait insuiuiional cm i- 
romiient that contributes to develop- 
ing and maintaining student success. 
This model is based on the premise 
that students are in the best position to 
define "success" for themselves and, 
therefore, for the institution and its 
programs. An important part of this 
model involves encouraging students 
to set high standards, documenting 
their goals and intended outcomes, 
and monitoring (heir progress toward 
achieving these outcomes. These stu- 
dent goals and intended outcomes 
form the basis for an extensive, institu- 
tion-wide assessment system that is far 
different from an assessment system 
devised from outcome measures and 
performance standards established bv 
such external groups as governing 
boards and accrediting agencies. 

Traditionally, some postsccondarv 
educators have operated on the prem- 
ise that students have the right to take 
anv class and to pass or fail that class, 
and some have even fell that a high 
failure rate is necessan to achieve high 
standards (Dressel, 1901). The student 
success paradigm operates from a 
vastly different perspective — that the 
school is obligated to devise a ioni se of 
study and a program of support serv- 
ices that enables students to meet their 
goals. 

Santa Fe Communis College 
(SFCC) has used the student success 
model to meet iis own high standards 
and, at the same lime, to demonstrate 
accountability to external bodies. The 
SFCC appioach to outcomes assess- 
ment connects student achievement 
directly to both the college's mission 
and specified learning objectives for 
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each discipline, course, and program. 
For example, course success and stud- 
ies of student achievement in math 
and Knglish courses are used to ensure 
that specified competencies are, in re- 
ality, being acquired while the student 
is in attendance. The degree of success 
experienced by students as measured 
by criterion-referenced exams be- 
comes a quality benchmark for both 
the college and the individual student. 
The college also measures student suc- 
cess after they leave the school. 

The following example illustrates 
how SFCC uses this assessment ap- 
proach. In 1989, 8f) percent of its stu- 
dents who took the national nursing li- 
censure exam (NC1.KX-RN) passed it; 
this was the same as the national aver- 
age. In 1990, while the national aver- 
age rose slightly 10 8G percent, SFCC's 
rate fell below 74 percent. This decline 
and a subsequent study led to a series 
ol recommendations designed to im- 
prove student performance on the lest. 
These included strengthening the cur- 
riculum in general, improving specific 
areas of weakness, enhancing student 
study skills, and improved monitoring 
of each student's academic perform- 
ance. In 1991, the College's pass rate 
on that test rose to 100 percent (Santa 
I*c Community College, 1991). 

Value-Added Assessment 

The concept of value-added can be 
applied whether one is assessing insti- 
lutional effectiveness, program quality, 
or student learning. The assessment 
strategy is to gather baseline informa- 
tion about entry-level competency and 
compare it with ex it -level perform- 
ance. This approach, which makes it 
possible to determine skills attained by 
students as they complete critical levels 
of a program, would be especially use- 
ful in determining program effective- 
ness for the many individuals who 
move in and out of college or into the 
job market without finishing a degree 
or certificate. Although value-added 
assessment has traditionally been con- 
ducted using a pre- and post-tesi de- 
sign, the potential for using alternative 
approaches such as performance as- 
sessmenis appears to be feasible. For 
example. Northeast Missouri Slate 



University, an early leader in (he appli- 
cation of value-added assessment to 
higher education, is now including 
performance assessment as part of in - 
value-added measurement. 

Alverno College, in Milwaukee. Wis- 
consin, uses multiple assessments 
across time not. only from the institu- 
tion's research perspective but from 
the student's perspective as well. One 
of Alverno's most widely recognized 
contributions to student outcomes as- 
sessment is the use of a matrix that de- 
fines abilities to be developed and 
demonstrated by students. 'Ib gradu- 
ate, each student must fully meet the 
criteria in each of the forty-eight ma- 
trix cells. Instruction is tied to these 
explicitly stated criteria. Assessment 
techniques at Alverno include written 
tests, verbal feedback from instructors, 
peer assessment, and sell-assessment. 
Through this process of continuous, 
multiple-method assessment of stu- 
dent performance, the College devel- 
ops a comprehensive record of perfor- 
mance for each student. 

One problem with value-added as- 
sessments is that they measure growth, 
which is not necessarily the same as 
competence. To an employer, the de- 
gree of change in competence is not 
likely to be important if a student has 
not met specified performance levels. 
However, Belcher (HKS7, p 32) points 
out that "measuring improvement 
does not replace the possibility of set- 
ting a floor by establishing exit stand- 
ards." Turn bull (H)87) argues that it is 
important to measure both growth and 
terminal competence. Measuring both 
is of critical concern where post secon- 
dary students enroll, gain skills, and 
move into the job market without com- 
pleting a degree or certificate pro- 
gram. J I ere, failure to address both as- 
pects of learning could lead to a situ- 
ation where an institution that is 
highly successful in producing a com- 
petent work force ( an appear to be in- 
efle< live because of a high non-com- 
pletion rate. 

Concept Mapping 

Once an institution commits to as- 
sessing outcomes and determining the 
value added to each student's skill and 
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knowledge, it needs a methodology for 
conceptualizing outcomes — thinking 
through the basic tenets and kev com- 
ponents of a program. A new approach 
that offers help in this area is concept 
mapping. This process enables diverse 
groups of practitioners to describe 
how their programs work and then, 
based on that collective wisdom, ar- 
ticulate the theory of the program, that 
is, how the program is intended to in- 
fluence outcomes. 

Practitioners of vocational educa- 
tion prescribe particular courses of ac- 
tion to achieve certain vocational edu- 
cation outcomes. In doing so, thev act 
as theorists: Behaviors, inter- 
pretations, and responses to situations 
represent their view of how vocational 
education programs are intended to 
function. Understanding the theoreti- 
cal underpinnings of a program is a 
practical step because ii is important :o 
understand bow programs can be ex- 
pected to work before the attempt to 
evaluate them is begun, Knowing the 
theory behind a program is essential to 
determining what to measure and 
what characteristics of a program are 
most important. 

Concept mapping employs small 
group processes, sorting and rating 
techniques, and multivariate statistical 
analysis techniques to represent the 
program in the form of a map. The 
map illustrates the participant stake- 
holders' ideas and how those ideas are 
interrelated. It also illustrates the ma- 
jor components of a program and can 
be used to indicate relationships 
among components and to show how 
these components are linked to out- 
comes. Once constructed, the map is 
then used by the participants as a com- 
mon framework to identify potential 
vocational education outcomes. Practi- 
tioner stakeholders who play a pari in 
the design and dclivcn' of a program 
should be* involved in the development 
ol this map. Thus, vocational educa- 
tors c an involve eclucatoi s, administra- 
tors, and Mudents as well as couimu- 
uitv, business, labor, or other repte 
sentatives in conceptualizing out- 
comes. 

Concept mapping can also be used 
to guide policymakers, managers, pro- 
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gram designers, and administrators in 
setting policy, conducting strategic 
planning, and facilitating program 
planning, implementation, and im- 
provement. Program theory illustrated 
through a concept map can also pro- 
vide feedback to those who created 
and funded a program. This type of in- 
formation can lead to improvements in 
policies that govern funding and im- 
plementation. 

Performance Assessment 

Performance assessment is an um- 
brella term that describes a variety of 
techniques designed to assess directly 
and "authentically" what students 
know and can do in a form other than 
paper and pencil or multiple choice 
tests. The assessments are considered 
more "authentic" because the student's 
master)' is demonstrated rather than 
inferred from his or her responses to 
questions about the subject. These as- 
sessments include profiles and records 
of achievement, portfolios, projects, 
and performance tasks. These tech- 
niques are particularly useful in meas- 
uring student outcomes that are diffi- 
cult to quantify with paper and pencil 
or multiple choice tests. Moreover, 
performance assessment techniques 
motivate students to learn — because 
thev are involved in the evaluation 
processes — and promotes the develop- 
ment of their own personal standards 
of quality. 

Although empirical evidence that 
performance assessment improves stu- 
dent outcomes is patchy, in both secon- 
dary and post secondary schools where 
performance assessment has gained a 
foe M hold, good news is reported. For 
example, in the Pittsburgh Arts PRO- 
PKl. schools, students creating portfo- 
lios of their writings reportedly write 
better, develop internal standards of 
quality, and think of themselves as pro- 
fessional writers (Camp, 1990). Other 
Mtcs offer significant evidence of im- 
portant and positive student outcomes 
(Archhald &■ Newmann, 1<M>2). 

Advocates of performance assess- 
ment hold that students cannot fully 
displa\ what thev know and can dor.v- 
<<//( in (espouse to motivating tasks 
(Raven, Interesting assessment 
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tasks calling for student-valued and 
complex learning skills not only pro- 
vide useful evaluations hut also posi- 
tive learning experiences. In this way, 
distinctions between assessment and 
learning are blurred; it is not merely 
that assessment informs future instruc- 
tion but that the assessment itself 'is in- 
stinctive (Camp, 1990). 

Since many performance assess- 
ment techniques are relatively new, 
educators are struggling with such is- 
sues as cost, faculty development, and 
public credibility. For example, al- 
though tasks that are intended to elicit 
student action and production arc di- 
rect and meaningful measures of stu- 
dents' knowledge and skills, develop- 
ers are finding it difficult and time- 
consuming to construct performance 
tasks in some areas. The specificity of 
the tasks, combined with the time re- 
quired to complete them, constrains 
the scope of what can be tested. 

Nonetheless, vocational educators 
have long used (asks, projects, and 
demonstrations to assess student 
achievement. Researchers and policy- 
makers can learn from vocational edu- 
cation's experience with performance 
assessment, while vocational educators 
can learn from the developing field of 
alternative assessments and adapt new 
strategies to their particular needs. 

Note: These different approaches 
have been presented separately to 
highlight what each can contribute to 
improving postsccondarv vocational 
education, but they are not either/or 
choices. These various approaches can 
be — and are in fact — used in connec - 
tion with one another. For example. 
TQM. the student success model, and 
performance assessment can all be 
used to assess the value added hv an 
educational program. And concept 
mapping would be useful in all eases to 
clarifv the theory of a program belore 
any assessment is planned. 
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